1. Field of the Invention
The present invention relates generally to the field of photography, and in particular to the manipulation of multiple images in order to isolate a particular subject of those images for use in electronic photo-compositing. More specifically, the present invention relates to a method for electronically processing multiple images of a foreground subject taken on different backgrounds in order to generate a new image of the foreground subject completely removed from the backgrounds, said new image of foreground subject being true in color and transparency for use in photo-realistic, composite images.
2. Description of the Related Art
A number of existing methods allow users of electronic imaging systems to extract a foreground subject from a scene in an image, isolating said subject from the background of the scene so that the subject may later be composited into a different scene in another image. However, existing methods have a number of deficiencies, including being mathematically inaccurate at determining correct colors and transparencies, requiring excessive user skill and attention, requiring costly, colored background materials, and generally producing results that are only marginally acceptable in many cases.
A first existing method requires the foreground subject to be photographed in front of a uniformly colored background so that a xe2x80x9cchroma keyxe2x80x9d method or xe2x80x9cblue screenxe2x80x9d method may be used to isolate the foreground subject from the uniformly colored background. An example of this method is described in U.S. Pat. No. 5,424,781. This method can yield acceptable results, but it commonly produces resulting images with fringes of color around the foreground subject""s edges, poor reproduction of small features, inaccurate color values in transparent areas of the subject, and it typically requires a high level of user skill and attention. In addition, a carefully chosen background color must be used because if the chosen background color appears anywhere in the foreground subject, this method will also assume that these colored areas are part of the background and will incorrectly treat these areas as transparency in the resulting mask even though the foreground subject may have been that color and fully opaque at that point.
A second existing method uses two images, a first image of a foreground subject on a background and a second image of the said background alone. In this method, a difference image is calculated from the first and second image to be reapplied to the first image to isolate the foreground subject from the background. Unfortunately, this method suffers from some of the same problems as the xe2x80x9cblue screenxe2x80x9d method above. If the foreground subject contains a color at a location in the image that is similar to the color of the background at that same location in the image, this method will interpret the color at this location as part of the background and will incorrectly assume it is the background, resulting in transparency in the mask even though the foreground subject may have been fully opaque at that point.
A third existing method allows the subject to be photographed on any background and electronically xe2x80x9ccutxe2x80x9d out of the background using software tools available in such digital image editing programs as Adobe Photoshop. However, if the foreground subject has more than a small amount of detail, it usually becomes a time-consuming and difficult process that yields poor results with high-contrast edges that do not blend smoothly in composite photographs.
The present invention is directed to overcoming the problems set forth above. This invention is a new and unique method for generating masks required for electronic photo-compositing of multiple images. The method described herein uses two images of a foreground subject to calculate an accurate mask, each image captured from the exact same viewpoint but each image having a unique background. By using two images of the same foreground subject on at least two different backgrounds, it is possible to accurately calculate the exact mathematical solution to the amount-of-transparency problem. In fact, this method can accurately calculate the transparency as multiple color channels of transparency, rather than as a single monochromatic channel as all existing methods do. This allows reproduction of the true color filter effects of the transparent areas of the foreground subject.
It is necessary to set realistic goals for the functionality of the present invention. Inconsistent lighting, automatic exposure cameras, and camera noise will always interfere with an ideal, fully automated solution, but it is possible to automate the most difficult part of generating a mask with edges that are free of fringes and artifacts. The resulting image and mask can be used in many popular imaging and animation software packages, usually with little or no additional editing required.
It is a goal of the present invention to provide a 100% true and accurate solution to the amount-of-transparency problem. Photographers and designers want to minimize the destructive effects of any image manipulation in order to maintain the original image fidelity. All previous methods used to generate masks have deficiencies such as hard edges, edges with fringes of unwanted colors, the inability to produce semi-transparent areas of the proper colors, and the inability to produce color filter effects. These deficiencies hinder the reuse of the resulting image because the image typically has visible artifacts that are unacceptable on all but a select number of backgrounds that minimize the visible artifacts. As the accuracy and fidelity of the masked image approaches the true transparency values, the visible artifacts are reduced or eliminated and the ability to reuse the image in multiple ways increases dramatically. With no visible artifacts or flaws, the image may be re-composited with any choice of backgrounds, or used in ways that other masking methods would not allow. This reuse has many benefits. The need to take and store multiple images, each tailored or made suitable for a particular application or use, is eliminated. It gives the designer greater freedom of design and the ability to make rapid design changes. It also reduces the amount of film and prints and the number of digitized images which must be stored, which in turn, saves money and resources.
It is a further goal of the present invention to solve the most difficult parts of mask generation in an automated way, drastically reducing the amount of editing required before the image can be used. It is very likely that future hardware and software improvements will provide the operator with even more advanced tools than are currently available to improve the software-generated mask.
It is a further goal of the present invention to minimize user effort. With a goal of reducing the overall time and skilled effort required to produce a finished image and mask, the entire production process was examined, from photography to final image placement. Current blue/green screen techniques and other image editing tools only focus on the post-processing, or editing stage that occurs after the images have been captured and digitally transferred into a computer. Understanding that additional information can be gathered at the photography stage that would virtually eliminate the need for post-processing or editing, the method described herein substitutes a small additional level of effort during photography to greatly reduce the amount of skilled effort required during post production. The result is a method that requires two unique photographs to be taken during the photography stage, but that requires little or no image editing before the resulting image with transparency mask can be used. Overall, the total amount of user effort, from photography to final image placement, is drastically reduced. Even though this method requires the capture of one additional image, the process of acquiring this additional image is rapid and does not place any new skill requirements on the photographer. In addition, the method does not introduce any new lighting techniques or equipment requirements, and it can assumed that photographers are already knowledgeable in lighting techniques and possess any equipment necessary to perform this method. If the photographer chooses, the actions of changing backgrounds and digitally capturing the two images can even be semi-automated, further reducing the workload in most cases. After the images are captured, most of the work is done for the operator by the software.
It is a further goal of the present invention to make the process as easy as possible for the photographer, even though the majority of the effort required to achieve a usable image and mask is shifted away from a post-processing stage to the photography stage. As mentioned above, this method does not introduce any new techniques or equipment requirements to the photographer. All of the skills and equipment required by this process are also required for everyday photography. The cost of specially colored paints and backgrounds required by blue/green screen techniques is eliminated since the present invention may utilize black and white backgrounds that can commonly be found in any photo studio. To simplify the lighting set-up, the present invention can provide compensation for unevenly lit backgrounds. It is not necessary to make the illumination of the background uniform, which helps save the photographer time and effort during set-up. The ability of an implementation of the present invention to interface with a digital camera and optional capture images electronically allows almost immediate review of the resulting image so that the photographer can judge the fidelity of the image on-the-spot. There is no need to wait for post-process editing to occur to see a final image. And by applying an automated-background changing device to an implementation of the present invention, the manual action of changing the background can be eliminated. The ability of the present invention to interface with a background changing device and a digital camera can produce a highly automated system that further reduces the effort required to capture images.
It is a further goal of the present invention to produce an image and mask solution in a compatible format suitable for use in popular imaging and animation software packages. The most widely used still-image processing software is Adobe Photoshop, and it is important that a compatible color space and file format be supported. Also supported is the export of raw color data with a matching mask, a format commonly used in non-linear video editing systems.
It is a further goal of the present invention to produce a mask solution that can reproduce color filter effects in photo-composite images. Current imaging and animation software packages can only utilize a single alpha channel and support simple transparency, but it is possible to reproduce the color filter effect in Adobe Photoshop using multiple layers with appropriate blending modes. It is likely that future imaging and animation software packages will make use of multi-channel masks to provide support for color filter effects. An implementation of the present invention may simply convert the multiple channel alpha values to a single alpha channel using a weighted sum of the color components to produce a luminance channel, similar to methods used in color to grayscale image conversion.